docs: hand off the three chronic Wolverine.RabbitMQ.Tests first-round failures (GH-3726) - #3727
Merged
jeremydmiller merged 2 commits intoJul 30, 2026
Merged
Conversation
… failures `CIRabbitMQ` reports success on main, but not on the first attempt. The same three tests fail round one and the flaky-retry harness rescues them, so the job goes green and nobody sees it. Found while merging #3722, which went red on CIRabbitMQ and looked like a regression. It was not: main @ 0a92b4c fails the identical three tests at nearly identical elapsed times, and the only difference was retry luck -- main's second round rescued all three, #3722's rescued two of three. #3722 touched no RabbitMQ code. multi_node_exclusive_listener_failover.listener_fails_over_when_the_leader_running_it_crashes Bugs.Bug_1594_ReplayDeadLetterQueue.can_replay_dead_letter_message(mode: BufferedInMemory) ConventionalRouting.end_to_end_...send_from_one_node_to_another_all_with_conventional_routing The first one matters most: it fails with "The exclusive listener agent never settled on a single surviving node -- it kept flapping", which is GH-3604's flap verbatim. GH-3604 is parked with PR #3610 as test-only on the premise that the flap is not reproducible on main. It reproduces on CI on essentially every run, so that premise needs revisiting. Doc carries the job IDs for both sides, the assertion text for all three, a suggested order of work, and the traps specific to this area (stateful broker baselines, retry-runs-in-isolation, `git stash` reverting to HEAD rather than main). No fix here -- evidence and hypotheses only, deliberately, so the next session starts from main with the diagnosis open rather than inheriting a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…andoff Re-ran #3722's failed CIRabbitMQ job on the same commit. Its first round failed the same three tests again, so this is 3 of 3 observed runs rather than "most of them": listener_fails_over_when_the_leader_running_it_crashes FAIL @ 1:37.15 Bug_1594...can_replay_dead_letter_message(BufferedInMemory) FAIL @ 4:24.39 ...conventional_routing.send_from_one_node_to_another... FAIL @ 10:02.72 A fourth joined intermittently, and failed twice inside that one job before passing: multi_node_exclusive_listener_recovery .rows_released_after_the_listener_is_already_running_are_still_recovered FAIL @ 8.91s, 15.56s It shares the multi_node_exclusive_listener_* family with the failover test, so the doc now says to investigate those two together rather than separately. The re-run concluded SUCCESS -- the harness logged "Flaky tests" and rescued all four. That is why #3722 was safe to merge at 31/31, and simultaneously the sharpest illustration of the problem: same commit, same three round-one failures, opposite job outcome purely on retry luck. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 29, 2026
Closed
erdtsieck
pushed a commit
to erdtsieck/wolverine
that referenced
this pull request
Jul 31, 2026
The handoff landed in JasperFx#3727 to carry evidence between sessions, and that work is done. Leaving it at the repo root now actively misleads: * its headline item — the exclusive-listener failover test — is fixed in JasperFx#3728, and its central claim that the listener "kept flapping" was wrong. The failover always worked; only the agent's reported status was stuck. * its Bug_1594 section describes a deterministic failure that does not exist. That test uses Servers.SqlServerConnectionString, and the three failing modes were a missing SQL Server container, not a product defect. JasperFxGH-3726 carries the corrected account of all four tests, and the two still-unexplained ones stay tracked there and on JasperFx#3729. Nothing is lost by deleting the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a handoff document at the repo root for #3726, so the diagnosis is on
mainrather thanliving only in a session transcript and a couple of expiring CI logs.
Docs only — no code, no test, no build change.
Why this exists
CIRabbitMQreports success onmain, but not on the first attempt. The same three tests failround one on essentially every run and the flaky-retry harness rescues them, so the job goes green.
It surfaced because #3722 went red on
CIRabbitMQand looked like a regression. It wasn't —main@
0a92b4cb7fails the identical three tests at nearly identical elapsed times, and the only thingthat differed was retry luck:
main's second round rescued all three, #3722's rescued two of three.The most significant item:
listener_fails_over_when_the_leader_running_it_crashesfails with"The exclusive listener agent never settled on a single surviving node — it kept flapping." That
is #3604's flap verbatim, and #3604 is parked with #3610 as test-only on the premise that the
flap is not reproducible on
main. It reproduces on CI on essentially every run, so that premiseneeds revisiting before anyone closes it out as a test problem.
What the doc contains
before they age out.
retry harness re-runs a failed test in isolation, so an isolation-sensitive test fails the retry
too;
git stashreverts to HEAD rather thanmain.Deliberately contains no fix. The root cause is not established, and this area has produced
wrong diagnoses from log-reading alone more than once. Evidence and ranked hypotheses only, so the
next session starts from
mainwith the diagnosis genuinely open.Closes nothing. Tracked by #3726.
🤖 Generated with Claude Code